IWPT 2020 Shared Task Results of clasp/004

Coarse F₁ Scores

Each score pertains to the combined test set of the language, without distinguishing individual treebanks and the enhancement types they annotate. The last line shows the macro-average over languages.

Language	Tokens	Words	Sentences	UPOS	XPOS	UFeats	AllTags	Lemmas	UAS	LAS	CLAS	MLAS	BLEX	EULAS	ELAS
Arabic	99.98	97.89	80.43	93.08	91.75	91.86	89.70	93.27	83.14	78.71	75.44	68.27	72.03	75.62	51.26
Bulgarian	99.93	99.93	97.14	98.68	96.35	97.59	95.75	97.29	93.36	90.20	86.82	83.69	83.60	87.72	84.90
Czech	99.90	99.90	94.11	98.08	93.54	93.02	91.56	98.21	90.57	87.21	84.98	77.52	83.40	83.44	67.13
Dutch	99.87	99.87	72.58	95.31	93.51	95.15	92.83	96.30	85.72	82.34	75.63	70.31	72.29	80.20	78.93
English	99.36	99.36	86.56	95.63	94.96	95.85	93.06	96.67	87.42	84.78	81.69	76.04	79.00	83.86	82.87
Estonian	99.74	99.74	91.79	96.65	97.54	95.19	93.66	95.63	85.36	82.33	80.84	75.75	76.68	79.43	60.44
Finnish	99.69	99.67	92.27	97.17	55.83	95.74	53.90	94.82	89.14	86.79	85.17	80.64	80.41	83.30	65.96
French	99.81	99.71	96.14	95.35	99.70	76.67	74.18	97.21	90.39	85.02	80.53	45.16	77.66	84.39	72.76
Italian	99.91	99.76	98.76	98.01	97.91	97.72	97.11	98.10	92.79	90.84	86.43	83.60	84.23	88.74	87.14
Latvian	99.82	99.82	99.01	96.02	88.25	93.45	87.72	95.54	87.84	84.44	82.16	73.91	78.25	80.60	66.01
Lithuanian	99.87	99.87	88.79	93.34	85.67	86.65	83.62	92.51	78.54	70.54	66.84	56.05	62.05	67.37	52.56
Polish	99.41	99.82	98.35	97.96	91.31	89.23	85.93	97.01	92.00	89.09	87.18	74.25	83.98	86.71	71.22
Russian	99.57	99.57	98.86	98.20	99.57	87.55	87.22	97.51	92.38	90.60	89.01	73.28	86.78	88.02	70.37
Slovak	99.97	99.97	90.93	96.36	87.17	91.59	86.36	94.73	89.96	86.16	83.76	74.54	78.47	83.31	65.16
Swedish	99.01	99.01	92.25	93.93	0.00	79.46	0.00	91.09	82.09	76.56	73.73	49.62	66.20	73.84	71.35
Tamil	99.58	91.42	95.08	82.60	78.80	81.89	78.10	85.14	61.23	55.76	53.43	46.40	49.61	55.32	42.15
Ukrainian	99.81	99.79	96.65	96.77	92.49	92.53	91.31	96.49	87.11	83.86	80.51	73.38	77.28	81.24	63.24
Average	99.72	99.12	92.34	95.48	84.96	90.66	81.29	95.15	86.41	82.66	79.66	69.55	76.00	80.18	67.85

Qualitative F₁ Scores

The system output for each language was split into parts corresponding to individual source treebanks. The scores then ignore errors in enhancement types for which the treebank lacks gold-standard annotation. The column LAvgELAS shows the qualitative ELAS averaged over treebanks of the same language; the final average in bold is averaged over languages rather than treebanks. For comparison, the last column then shows the coarse ELAS F₁ for the given language.

Treebank	Tokens	Words	Sentences	UPOS	XPOS	UFeats	AllTags	Lemmas	UAS	LAS	CLAS	MLAS	BLEX	EULAS	ELAS	Enhancements	LAvgELAS	cf.
Arabic-PADT	99.98	97.89	80.43	93.08	91.75	91.86	89.70	93.27	83.14	78.71	75.44	68.27	72.03	75.77	51.36	(no xsubj)	51.36	51.26
Bulgarian-BTB	99.93	99.93	97.14	98.68	96.35	97.59	95.75	97.29	93.36	90.20	86.82	83.69	83.60	87.72	84.90		84.90	84.90
Czech-FicTree	99.98	99.93	92.29	96.20	76.15	78.96	72.13	97.60	88.71	83.71	80.30	52.88	77.93	79.61	67.95		65.80	67.13
Czech-CAC	99.99	99.98	97.12	97.40	92.98	92.09	90.68	97.22	90.54	86.78	84.26	76.61	82.14	82.18	63.44		65.80	67.13
Czech-PDT	99.97	99.97	93.98	98.50	95.38	94.61	93.67	98.55	90.99	87.98	85.89	80.21	84.45	84.24	67.60		65.80	67.13
Czech-PUD	99.15	99.15	95.85	96.20	92.27	91.37	89.83	96.15	88.31	83.49	80.59	73.01	78.70	80.18	64.20	(no codepend)	65.80	67.13
Dutch-Alpino	99.96	99.96	89.98	96.33	94.76	96.28	94.13	96.97	89.56	86.44	81.22	75.76	77.80	84.39	83.18		79.03	78.93
Dutch-LassySmall	99.78	99.78	58.55	94.34	92.32	94.07	91.59	95.66	82.05	78.42	70.47	65.28	67.20	76.22	74.88		79.03	78.93
English-EWT	99.00	99.00	81.16	95.39	95.11	96.09	93.89	97.20	86.22	83.59	80.20	76.00	78.49	82.71	81.92		82.95	82.87
English-PUD	99.78	99.78	97.22	95.91	94.78	95.56	92.09	96.05	88.85	86.20	83.58	76.09	79.65	85.21	83.98		82.95	82.87
Estonian-EDT	99.96	99.96	93.32	97.18	98.04	95.77	94.42	96.04	86.69	83.82	82.40	77.61	78.31	81.46	61.82	(no xsubj)	58.55	60.44
Estonian-EWT	98.65	98.65	82.67	94.08	95.19	92.42	89.99	93.65	79.01	75.20	73.26	66.72	68.77	71.57	55.29	(no codepend, no xsubj)	58.55	60.44
Finnish-TDT	99.77	99.73	93.05	96.97	97.72	95.27	94.35	94.98	88.62	86.18	84.66	79.61	80.24	82.68	65.60		66.16	65.96
Finnish-PUD	99.58	99.58	91.09	97.43	0.00	96.37	0.00	94.61	89.82	87.61	85.84	82.00	80.64	84.52	66.72	(no codepend, no xsubj)	66.16	65.96
French-Sequoia	99.58	99.37	75.42	96.52	99.37	78.37	77.23	97.51	85.08	81.96	79.79	48.38	78.11	80.70	78.28	(no gapping, relcl, case)	80.62	72.76
French-FQB	99.91	99.85	99.93	94.87	99.83	75.96	72.92	97.08	92.61	86.29	80.84	43.82	77.47	85.89	82.96	(no gapping, relcl, case)	80.62	72.76
Italian-ISDT	99.91	99.76	98.76	98.01	97.91	97.72	97.11	98.10	92.79	90.84	86.43	83.60	84.23	88.74	87.14		87.14	87.14
Latvian-LVTB	99.82	99.82	99.01	96.02	88.25	93.45	87.72	95.54	87.84	84.44	82.16	73.91	78.25	80.60	66.01		66.01	66.01
Lithuanian-ALKSNIS	99.87	99.87	88.79	93.34	85.67	86.65	83.62	92.51	78.54	70.54	66.84	56.05	62.05	67.37	52.56		52.56	52.56
Polish-LFG	97.68	99.86	98.55	97.66	83.22	72.85	61.40	97.28	90.97	86.20	85.13	51.25	82.31	85.52	72.97	(no gapping)	71.33	71.22
Polish-PDB	99.87	99.83	98.39	98.32	94.04	93.96	92.86	97.29	92.34	90.06	88.17	81.03	85.25	87.33	71.49		71.33	71.22
Polish-PUD	99.80	99.78	97.91	97.53	92.09	92.25	90.74	96.30	92.10	89.36	86.81	77.98	82.89	86.40	69.53		71.33	71.22
Russian-SynTagRus	99.57	99.57	98.86	98.20	99.57	87.55	87.22	97.51	92.38	90.60	89.01	73.28	86.78	88.02	70.37		70.37	70.37
Slovak-SNK	99.97	99.97	90.93	96.36	87.17	91.59	86.36	94.73	89.96	86.16	83.76	74.54	78.47	83.31	65.16		65.16	65.16
Swedish-Talbanken	99.23	99.23	90.87	94.28	0.00	84.71	0.00	91.83	81.30	75.49	72.95	54.81	66.62	72.86	70.81		71.37	71.35
Swedish-PUD	98.78	98.78	93.87	93.55	0.00	73.84	0.00	90.30	82.92	77.70	74.56	44.07	65.76	74.89	71.93		71.37	71.35
Tamil-TTB	99.58	91.42	95.08	82.60	78.80	81.89	78.10	85.14	61.23	55.76	53.43	46.40	49.61	55.39	42.19	(no gapping, xsubj, relcl)	42.19	42.15
Ukrainian-IU	99.81	99.79	96.65	96.77	92.49	92.53	91.31	96.49	87.11	83.86	80.51	73.38	77.28	81.24	63.24		63.24	63.24
Average	99.60	99.29	91.67	95.78	82.54	89.70	78.17	95.46	86.89	83.13	80.19	68.44	76.61	80.60	69.20		68.16

IWPT 2020 Shared Task Results of clasp/004

Coarse F1 Scores

Qualitative F1 Scores

Coarse F₁ Scores

Qualitative F₁ Scores